Using Experiments to Improve Ideal Point Estimation in Text with an Application to Political Ads∗

Researchers are rapidly developing new automated techniques to scale political speech on an ideological dimension. Yet, the task has proven difficult across many settings. Political advertisements, in particular, have eluded such efforts. Candidates air relatively few ads, containing limited policy information, and there is little agreement about how to model political speech in the campaign, much less in general. Rather than model the underlying ideological structure of words, I develop an experimental approach to directly measure the content of political ads. I randomly assign ads to subjects, recruited in a large-N survey, who are asked to guess the party (or ideological leaning) of the featured candidate. Ads are then scaled as their expected partisan guessing score. This score is well-measured given random assignment and subject recruitment, and can be used in a supervised learning approach to scale other ad text. Due to the inferential nature of the task, subjects are less likely to exhibit bias in their guessing. Further, I show that the average partisan signal in ads is synonymous with an ideological dimension in the minds of respondents. I implement a number of tests to assess party guessing as a way to scale ads, each of which indicate remarkable reliability and validity in the approach. Finally, I explore ways to scale up the guessing task to a much larger set of ads. Beyond scaling ads, the inferential approach outlined here can be generalized to measure a much wider array of dimensions contained in speech and text data.